Unimodal and Multimodal Representation Training for Relation Extraction

Abstract Multimodal integration of text, layout and visual information has achieved SOTA results in visually rich document understanding (VrDU) tasks, including relation extraction (RE). However, despite its importance, evaluation the relative predictive capacity these modalities is less prevalent. Here, we demonstrate value shared representations for RE tasks by conducting experiments which each data type iteratively excluded during training. In addition, text are evaluated isolation. While a bimodal approach performs best (F1 = 0.684), show that most important single predictor entity relations. Additionally, geometry highly may even be feasible unimodal approach. Despite being effective, highlight circumstances where can bolster performance. total, our efficacy training joint RE.